Skip to content

Add opt-in LISTEN/NOTIFY long polling - #41

Open
Thom Chiovoloni (thomcc-work) wants to merge 12 commits into
mainfrom
thomcc/long-polling
Open

Thom Chiovoloni (thomcc-work) wants to merge 12 commits into
mainfrom
thomcc/long-polling

Conversation

@thomcc-work

@thomcc-work Thom Chiovoloni (thomcc-work) commented Oct 20, 2026 •

Copy link
Copy Markdown

Summary

This is part of an effort to make duroxide-pg-opt redundant, although it stands on its own and has its own benefits, so I'm PRing it separately. It's a draft until I finish AI review.

Adds notification-driven long-polling for orchestration and activity work, adapted from duroxide-pg-opt. Existing constructors continue to use short polling by default. The goal is to reduce empty-queue queries and polling-induced latency without changing queue eligibility, locking, or acknowledgment semantics.

let mut config = ProviderConfig::url(database_url);
config.long_poll.enabled = true;
let provider = PostgresProvider::new_with_config(config).await?;

The same configuration is supported for Entra-authenticated connections.

Implementation

  • Register notification waiters before checking the queue to avoid lost wakeups.
  • Track upcoming work in both queues, including delayed inserts and abandoned/released work.
  • Rescan periodically to recover missed notifications, and resubscribe after detected connection failures.
  • Bound timer storage and coalesce catch-up scans and repeated unsuccessful wake-triggered fetches to a 100 ms cadence. The first notification wake remains immediate.
  • Preserve zero-timeout fetch behavior. The polling deadline bounds notification waiting; it never cancels a potentially mutating database fetch.
  • Own listener/timer task lifetimes and safely discard cancelled refresh connections.
  • Use table-identity-based channel names, avoiding PostgreSQL's notification-channel length limit.
  • Require SQLx 0.8.3 for explicit listener reconnect handling; native-TLS configuration is unchanged.

Migration and activation

Release dependency: do not release this PR before #32 and #51. A release containing this PR's migration 0030 must also contain #32's migrations 0027-0028 and #51's migration 0029. Likewise, #51 must not be released before #32. Shipping a later migration without its predecessors creates a ledger that a later combined build rejects as a non-prefix; coordinate the release order rather than editing migration records.

Migration 0030 installs disabled notification triggers. Notifications are activated schema-wide only when a provider is constructed with both MigrationPolicy::ApplyAll and long-polling enabled. Its cleanup function preserves migration 0028's batched-pruning cleanup. Disposable databases that ran an earlier draft of this PR as migration 0027 must be recreated.

VerifyOnly performs no DDL and requires prior activation. Activation is atomic, follows acknowledgment lock ordering, and retries bounded lock contention. Disabling an individual provider's listener does not disable shared triggers.

Long-polling reserves one connection within the configured pool, so at least two connections are required. Client-only providers can leave it disabled.

This remains a normal forward migration: older binaries reject unknown migration versions even when notifications are disabled. It does not convert databases using the incompatible duroxide-pg-opt migration history.

Performance validation

Ignored benchmarks cover idle SQL, sparse/loaded workflows, and notification-lock contention with a synchronous standby. The README documents their settings and statistics-role requirements. Run comparisons on otherwise idle, controlled hosts; no performance conclusions are claimed from the earlier runs on a shared, busy machine.

Operational considerations

  • Activated queue-writing transactions cannot use PREPARE TRANSACTION. Schemas that have not opted in retain their previous behavior.
  • PostgreSQL serializes notification-emitting commits through a server-wide lock. WAL or synchronous-replication stalls can delay other notification writers, including in other databases. Loaded-write performance should be evaluated before activation.
  • Detected disconnects recover automatically, but silent network partitions are not actively probed. Polling fallbacks remain available; SQLx listener cleanup can still wait indefinitely on a wedged socket. Proactive detection and bounded cleanup are deferred.
  • Providers must be recreated if their queue tables are dropped and recreated.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved wakeup gaps and schema-wide listener recovery require fixes and PostgreSQL-backed human validation.

2 open issues
What changed in this PR

Adds opt-in notification-driven long polling to the PostgreSQL provider while preserving short polling by default.

Changes:

  • Adds listener recovery, bounded timer tracking, and long-poll configuration.
  • Installs disabled notification triggers with explicit schema-wide activation.
  • Adds integration coverage, CI validation, and operational documentation.
File Description
tests/​postgres_provider_test.rs Enables validation in either polling mode.
tests/​long_poll_tests.rs Adds integration tests and an idle-query benchmark.
src/​provider.rs Integrates configuration and long-poll fetches.
src/​notifier.rs Implements listeners, timers, recovery, and waiting.
src/​migrations.rs Adds notification activation and verification.
src/​lib.rs Exports and documents long-poll configuration.
README.md Documents usage and operational limitations.
migrations/​README.md Documents activation and test commands.
migrations/​0027_diff.md Records migration changes.
migrations/​0027_add_long_poll_notifications.sql Installs disabled triggers and updates cleanup.
docs/​migration-policy.md Explains activation and compatibility.
CHANGELOG.md Records the unreleased feature.
Cargo.toml Raises the SQLx minimum to 0.8.3.
.github/​workflows/​test.yml Runs validation with long polling enabled.

Review effort: Balanced

Comment thread src/notifier.rs Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Schema-wide trigger activation and concurrency-sensitive listener, timer, and cancellation behavior warrant human review backed by PostgreSQL integration results.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

- Future-dated lease hints, including session expiry, plus immediate release hints for delayed abandonment.
- Warm fetch connections, bounded timer scans, and independent degraded-mode fallback.
- Bounded, jittered migration/activation retries with rollback and deadlock regressions.
- Updated trigger inventory, deterministic catch-up coverage, and a reproducible benchmark driver.
Run timer reads in transactions with server-side lock and statement timeouts. Retain one owned scan across client watchdog wakeups and listener reconnects instead of cancelling and replacing an in-flight query.

Add the supplied backend-accumulation regression, connection-reuse and single-flight coverage, and a timer wake deadline that cannot pass through the polling fallback. Addresses review items 41-N1 and 41-N5.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Place long-poll notification DDL after the pending 0027-0029 migrations, preserving migration 0028's prune_executions_batch cleanup and companion diff baseline. Update upgrade tests and documentation without stacking the prerequisite PRs.

Include SQLSTATE and the full context chain in DDL retry warnings, test those diagnostics, and document draining writers and retrying ApplyAll when the fixed contention budget expires. Validated the intended predecessor sequence in an isolated source copy. Addresses 41-C1, 41-N3, and 41-N4.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a 16+16 dispatcher, 64-client profile with sampled notification-commit and synchronous-replication wait time. Exercise workflow claims and saturated orchestration, activity, and session renewals with verified synchronous-standby and pool-capacity preconditions.

Document notification-bearing operations and reproducible benchmark methodology without publishing performance results from uncontrolled runs. Include timer transaction control in idle SQL counts. Addresses review item 41-N2.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the database message in ordinary migration error display while retaining the SQLx source for SQLSTATE retry classification. Add a red/green regression that also proves failed migration statements leave the ledger, stored function, and durable rows unchanged.

State that migration 0030 must not ship before the 0027-0029 prerequisites, including the dependency of #51 on #32. Addresses review items 41-N6 and 41-N7.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep timer scans single-flight, but queue a fresh catch-up for both queues when a reconnect adopts an older snapshot. Run scan transactions READ ONLY and verify rejected writes preserve durable state and connection defaults.

Replace SQL-lock timing gates with controlled acquisition gates, verify successful commits rather than idle backends, and wait through reconnect re-entry before checking the scan count. Add the disconnect-gap regression and benchmark statistics-role preflight/documentation. Addresses 41-N8 and all 41-N9 nits; no performance figures are published.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep close-on-drop armed for migration failures, unconfirmed unlocks, and cancellation, but explicitly return the healthy connection to SQLx after success and confirmed advisory-lock release. This leaves short-poll startup with its original warm connection.

Add backend-identity and immediate-idle assertions for migration/no-op/activation success, plus failed-unlock and cancellation safety regressions. Update the migration lifecycle documentation. Addresses review item 41-N10.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match #51's associated load_migrations and parse_version functions, including the failed-unlock regression added by #41. This keeps the standalone branch buildable and avoids E0599 when the prerequisite changes are combined.

Retain b33a3f2's connection-reuse behavior without changing pool sizing or tuning for the upstream barrier-based test. Addresses review item 41-N11.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Guard migration connections with RAII: unsuccessful exits mark the connection for closure before its field is dropped, while successful migration and confirmed unlock allow SQLx's normal asynchronous pool return.

Keep the same-backend and advisory-lock regressions, remove the immediate-idle assertion, and document asynchronous return. No pool sizing changes or timing-test workarounds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep both delayed items enqueued up front, but spawn the second fetch without asserting that it is still waiting. A slow dispatcher may start that fetch after the second item is already eligible.

Exercise both queues with immediate and deliberately late second fetches. The added late-start case reproduces the old assertion failure and passes after the fix; the shared waiting helper and production code remain unchanged. Addresses review item 41-N12.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants